This Page is Inserted by IFW Iniexlffif mi §€St»M»g 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document me accia: ate repres^tatiof^ ef tiie 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ black BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES- ^ 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMIMT 

□ KEiTERENCECS) OR EXHIB1T<S) SUBMITTED ARE POOR QUALIT¥ 

□ OTHER: . — 

IMAGES ARE BEST AVAILABLE COPY. _ 

As rescanaing these documents will not correct the mage 

problems checked, please doMt report these problems to 

the IFW Image Problem Mailbox. 



(19) 




Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 




(12) 



(11) EP 0705 025 A2 

EUROPEAN PATENT APPLICATION 



(43) Date of publication: 

03.04.1996 Bulletin 1996/14 

(21) Application number: 95420266.9 

(22) Date of filing: 25.09.1995 

(84) Designated Contracting Slates 
DE FR GB 

(30) Priority: 30.09.1994 US 315781 



(71) Applicant: EASTMAN KODAK COMPANY 
Rochester, New York 1465Q-2201 (US) 



(51) Intel 6: H04N 1/40 



(72) Inventor: Rabbant, Majid, 

c/o Eastman Kodak Company 
Roctiester, New York 14650-2201 (US) 

(74) Representative: Buff, Michel et al 
Kodak-Pathe 

Departement des Brevets ot Licences CRT 
Centre de Recherches et de Technologie 
Zone Industrielle 

F-71102 Chalon sur Sadne Cedex (FR) 



CM 
< 

CM 
O 

ID 
O 
1^ 

O 

Q. 
UJ 



(54) Method and apparatus for reducing quantization artifacts in a hierarchical image storage 
and retrieval Rvjstf^m ^ ^ 



and retrieval system 

(57) The system and method reduces the quantiza- 
tion artifacts in the addition and removal of a digital wa- 
termark to and from a selected resolution image of a hi- 
erarchical image storage system where the watermark 
removal record is placed in a higher resolution image 
component- For those applications where presenting the 
image quality of a higher resolution image component is 
more critical than preserving the image quality of a tower 
resolutton image component, the low-resolution image 
is modified according to the teachings erf the present in- 
vention to minimize and in many cases eliminate the 
quantizatton artifacts at the higher resoiutkxi compo- 
nent. 




Printed by Jouve (FR). is. rue SalntOnls. 75001 F%R|S 



1 



EP0 705 025 A2 



2 



Description 

Field of The Invention 

The present invention is related to the field of digital 
image processing and more particularly to methods and 
associated apparatus for reducing quantization artifacts 
in a hierarchical image storage and retrieval system that 
employs the quantization of residual image components. 

Background Of The Invention 

Image pyramids provide an efficient means for the 
storage and retrieval of digital images at various resolu- 
tions. It is customary in such hierarchical image storage 
and retrieval systems to construct a multiple of image 
resolutions by repeatedly filtering and subsampling a 
high resolution image using a scheme similar to the 
teachings found in the patent to Melnychuck and Jones 
(U. Pat. No. 4,969.204). In such cases, the higher res- 
olution components are usually stored in the form of 
quantized residuals that are entropy encoded to save 
storage space. In the context of the present invention, 
we will make reference to one such system, namely the 
Kodak Photo CD system, as an example to demonstrate 
the fundamental concepts underlying the invention. 
However, it shoukJ be noted that this example is only 
used to give the reader an insight as to the operation of 
one or more embodiments of the invention with the un- 
derstanding that other hierarchical image storage and re- 
trieval systems with other resolutk>ns or arrangements 
that store the high resolutbn components as reskiuals 
may be chosen to suit specific needs without detracting 
from the teachings of the present invention. 

The Kodak Photo CD system is composed of an im- 
age hierarchy or pyramid in which the hi^est resolutrcn 
image contains 3072 x 2048 pixels and is referred to as 
the 16BASE image. This resolution is in nrx>st cases ad- 
equate to produce photographic quality originals on an 
appropriate digital output device. The next level of hier- 
archy, referred to as the 4BASE, is composed of 1536 x 
1024 pixels and is adequate to generate a high quality 
HDTV display, or a small-sized photographic quality print 
on an appropriate digital output devk^. The tower reso- 
lutkxi levels are, respectively, the BASE innage consist- 
ing of 768 x 512 pixels (suitable for NTSC/PAL/SECAM 
televiskxi viewing), the BASE/4 image consisting of 384 
X 256 pixels (suitable for sub-NTSC displays), and the 
BASE/1 6 consisting of 1 92 x 1 28 pixels (suitable for dis- 
playing thumbnail images). The complete image hierar- 
chy is constructed from the 16BASE image using the 
teachings found in the #204 patent to Melnychuck and 
Jones prevtously cited. The BASE/16. BASE/4, and the 
BASE images are stored onto a digital storage medium 
such as a CD in uncompressed form while the 4BASE 
and 16BASE images are represented as residuals and 
are quantized and compressed by using entropy coding 
prtor to their storage on the CD. 



A quantizer is a many-to-one mapping and. as such, 
the quantized data often contains degradations com- 
pared to the original unquantized cJata. The degradations 
can be particularly significant when the input data has a 
5 distributton (histogram) containing tong tails, as those 
values are clipped to the outermost quantization level. In 
a hierarchical innaging system such as the Kodak Photo 
CD system, the residual inr^ge usually has a symmetric 
distribution with a high peak and decaying tails, some- 
10 times approximated by a Uplacian (two-sided exponen- 
tial) distributton. The minimum-nnean-squared-error 
quantizer that accomnxxlates such a distribution con- 
tains dense inner level (in the region where the signal 
peaks) and sparse outer level (around the tails). /Vn ex- 
15 ample of how a quantizer processes treats signal values 
is shown in the table erf Fig. 10. It is based on the principal 
that by introducing a larger error in those values where 
the signal occurs infrequently and a smaller error at val- 
ues where it occurs more frequently, the overall average 
20 error can be minimized. 

/\s described in the teachings of the prevtously cited 
patent to Melnychuck and Jones, the constmction of a 
high-resolutton image is accomplished by adding the 
quantized resrclual to the interpolated tow-resolution 
25 component from the previous stage. Thus, the error re- 
sulting from the quantizatton of the residual is propagat- 
ed to the higher-resolution images. Although the residu- 
als with large magnitudes occur infrequently, it shou W be 
noted that they often represent high-contrast edges or 
30 details in an image, and the introduction of a large quan- 
tization error can create an annoying visual artifact or 
toss of detail in the high-resolution image. One of the ob- 
jectives of the present inventton is to introduce a method 
whereby the tow-resolution image whose image quality 
35 is less critical can be nrxxiified in order to minimize the 
quantizatton errors in the high-resolutton image. 

The aforementtoned hierarchical storage technique 
is combinable with the insertkxi and removal of a digital 
watenmaric according to the teachings of 
^ EP-A-94420293.6byRabbanietal..toformadistributed 
image system In a distributed image system it is com- 
mon to deliver an image d compromised image quality 
for the purposes of browsing or proofing. The compro- 
mise can be made with the use of a removable water- 
45 nr^ark. Primary use of such a system is by professtonal 
portrait photographers who distribute a plurality of imag- 
es to a customer for selectton and approval. The water- 
maric is a form of graphic overlay that nnay contain a cop- 
yright notice or information regarding the restricted use 
^ of the image. 

Upon selection of the desired innage by the custom- 
er, the professional delivers a high quality renditton <rf the 
innage either digitally or in hard copy f omn without the wa- 
termark. At all times the professional possesses the sole 
55 means of generating and delivering unmart^ed high qual- 
ity images. In a conventtona! photographto system the 
means for generating wouW be the original negatives of 
the images; in a digital hierarchical system according to 
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the invention of Rabbani et al., the means tor ^nerating 
are higher resolution residual components. 

In a digital imaging system, and in particular one that 
includes a hierarchical form of digital storage and retriev- 
al, the professional may use a suitable digital storage 
medium such as a CD for the distribution of proofs. In an 
unrestricted environment, the customer may choose a 
desired image resolution from the hierarchy for the pur- 
poses of browsing, proofing, or hard copy fulfillment. In 
those instances where it is desirable for the professional 
to deliver the digital storage medium containing the en- 
tire image hierarchy to the customer, it is also most eco- 
nomical to record the entire image hierarchy once onto 
the digital storage medium and avoid having to make a 
second copy containing only low resolution components 
for distribution. However, rt is also desirable to restrict 
the use of selected high resolution components for the 
purpose of full image quality fulfillment until payment has 
been received. 

The invention of the Rabbani et al. application, pre- 
vbusly referenced, teaches a method whereby the pro- 
fessional places a digital watermark on a selected image 
component. The removal of the watemiark is done 
through an additional image component containing the 
reverse of the watermark. The customer, having posses- 
sbn of the CD storage medium, woukJ possess the 
means for generating his own high quality hard copy 
when authorized by the professional. 

When a watermark is applied to a tow-resolution im- 
age, the pixel values in that regk)n of the low-resolutton 
image affected by the watermark can become complete- 
ly different from the pixel values in the high-resolution 
image corresponding to the same region. As a result, in 
those regions, the difference behween the interpolated 
watermarked image arxJ the high-resolutkxi image can 
become fairly large whk^h, given the nature of the quan- 
tizer, can result in large quantizatkxi errors. These errors 
often appear as a visual trace of the watermark in the 
high-resolutwn image and significantly impair the visual 
quality of the high-resdutbn image. Minimizalkxi of such 
errors is addressed by the present inventk)n. 

Summary of the Invention 

One embodiment of the present inventton is a meth- 
od for reducing quantizatkxi artifacts in a hierarchcal im- 
age system of the type that decomposes a high resolu- 
tton image into tow resolutton Image components, includ- 
ing tow resolution image pixel values, and reskiual image 
pixel values comprising the steps of:: 

a) forming a predicted higher resolutton image by 
interpolating the pixel values of the low resolutton 
image components; 

b) determining the differerK:e values between the 
pixel values in the low resolution image and the cor- 
responding pixel values in the higher resolutton 
image; 



c) comparing each of the difference values against 
quantizer conversion values to detemnine if an error 
would exist for an executed conversion; 

d) adjusting the image pixel values within the tow 
5 resolution image component if an error would exist, 

to a value that eliminates the error to form an 
adjusted low resolution image; and 

e) using the unadjusted image pixel values as part 
of the adjusted low resolution image component if 

10 an error does not exist. 

Another embodiment of the present invention is a 
method for reducing quantization artifacts in a hierarchi- 
cal image system of the type that decomposes a high 
^5 resolution image into tow resolutton innage pixel values 
and residual image pixel values and applies a waterrriark 
to the low resolutton image pixel values comprising the 
steps of: 

20 a) applying watermark pixel values to pixel positions 
within the tow resolution image to form a water- 
marked tow resolution image component; 

b) determining the difference values between the 
pixel values in the watermarked low resolutton 

^5 image and the corresponding pixel values in the 
higher resolutton image; 

c) comparing each of the difference values against 
quantizer conversion values to determine if an error 
would exist for an executed conversion; 

30 d) adjusting the watermark image pixel values within 
the watemr^rked low resolution image component if 
an error would exist, to a value that eliminates the 
error to form an adjusted watermarked tow resolu- 
X\on image; and 

3S e) using the watermark image pixel value as part of 
the adjusted watermarked low resolution image 
component if an error does not exist. 

From the foregoing it can be seen that it is a pre- 
40 ferred object of the present invention to prcvtoe a method 
and associated apparatus for improving the quality of a 
high-resolutkxi image component by nnodifying a low- 
er-resolutton image component whose image quality is 
less crittoal. 

^ It is arx>ther object dt the present inventton to im- 
prove the quality of a high-resolutk>n image component 
by modifying a tower-resolutton image component 
whose image quality is less critical when a watermark is 
inserted and removed. 

^ Yet another object of the present invention is the re- 
ductton of quantization artifacts in the addition and re- 
moval of a digital watermark to and from a selected res- 
olution image where the watermark removal record is 
placed in a higher resolutton image component. 

^ The above and other objects of the present invention 
will become more apparent when taken in conjunctton 
with the foltowing description and drawings wherein like 
characters indicate like parts and which drawings form 
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8 part of the present invention. 

Brief Description Of The Drawings 

Figure 1 iliustrates, in blocic diagram form, a PRIOR s 
ART hierarchical image decomposition scheme com- 
bined with a scheme for inserting a digital watermark in 
a low-resolution image component and placing a water- 
mark removal component in a high-resolution residual. 

Figure 2 illustrates, in block diagram form, a PRIOR io 
ART hierarchical image reconstruction scheme for re- 
covering a high-resolution innage that has been decom- 
posed by the scheme illustrated in Figure 1 , 

Figure 3 is an example of the brightness component 
of a BASE image in pixel value form. is 

Figure 4 is an example of a watermarked BASE im- 
age in pixel value form. 

Figure 5 is an example of the pixel values of a 
4BASE image corresponding to the pixel values of the 
BASE image of Figure 3. 20 

Figure 6 illustrates the relative X,Y position of the 
pixels used in a linear interpolation scheme. 

Figure 7 illustrates the interpolated watermarked 
BASE image pixel values corresponding to the pixel val- 
ues of the watermarked BASE image of Figure 4. 25 

Figure 8 illustrates the pixel values of a residual im- 
age formed as the difference between the images in Fig- 
ures 5 and 7. 

Figure 9 illustrates the pixel values of the residual 
image of Rgure 8 after quantization. 30 

Figure 10 illustrates, in table form, the quantizer in- 
put<mtput characteristk^s used in generating the quan- 
tized reskJual image pixel values of Figure 9. 

Figure 11 illustrates the pixel values of the recon- 
structed 4BASE image using the PRIOR ART schemes 3S 
of Figures 1 and 2. 

Figure 12 illustrates the improvement to the PRIOR 
ART system of Figure 1 . 

Figure 13 illustrates, in a flow diagram form, the 
steps necessary for the adjustment of the values of the 40 
k>WHresolulk)n inriage pixels during the first pass of the 
present ufivernkxi. 

Figure 14 illustrates the pixel values of a modified 
watemfiarked BASE image resulting from a first pass of 
the present invention. 45 

Figure 1 5 illustrates the pixel values of a reconstruct- 
ed 4BASE image using the pixel values of the modified 
watermarked BASE image of Figure 14. 

Figure 16 is a flow diagram of the procedure em- 
ployed in STEP 1 of the second pass of the present in- so 
vention to adjust the pixel values of the low-resolutkxi 
image. 

Figure 17 is a flow diagram of the procedure em- 
ployed in STEP 2 of the second pass of the present in- 
vention to adjust the pixel values of the low-resolutk>n ss 
image. 

Figure 18 is a ftow diagram of the procedure em- 
ployed in STEP 3 erf the second pass of the present in- 



ventk>n to adjust the pixel values of the tow-resolution 
image. 

Figure 19 illustrates the pixel values of a modified 
watermarked BASE image resulting from a second pass 
of the present inventkxi. 

Figure 20 illustrates the pixel values of a reconstruct- 
ed 4BASE image using the watermarked BASE image 
of Figure 1 9. 

Detailed Description Of The Invention 

The present invention generally addresses the po- 
tential problem of artifact generation in a high resolution 
image caused by the quantizatkx) of residuals. One ex- 
ample is when the high resolutkxi image contains high 
contrast edges or areas of detail that can give rise to a 
large reskJual value. Consequently, the quantizatbn or 
clipping of this residual value can degrade the quality of 
the high resolution image reconstructed from this resid- 
ual. 

Another example is when a watermark is inserted 
into the tow resolution image. In this case, the pixel val- 
ues in that region of the tow-resolution innage affected by 
the watermark can become completely different from the 
pixel values in the high-resolution image corresponding 
to the same region, thus resulting in targe residual val- 
ues The error resulting from the quantization of such 
large restoual values often appears as a visual trace of 
the watermark in the high-resolution image. 

A PRIOR ART watermark system arrangement is il- 
lustrated in Figure 1. This arrangement is the same as 
the arrangement illustrated in Figure 2 of US. Patent Ap- 
plication Serial No. 146,371, referenced above. In Box 
34, a watermark W is first inserted in the BASE image I 
to arrive at a watemnarked BASE image ly^. This water- 
marked BASE image is then interpolated to the size of 
the 4BASE image in interpolator 24 using linear interpo- 
lation. A difference is formed between the original 
4BASE image taken from the output of a decompositton 
apparatus 101 and the interpolated watermarked BASE 
image in subtracter 32 to form a modified 4BASE reski- 
ual that senses as the watermark removal record. This 
modified 4BASE reskiual is encoded by encoder 20, en- 
crypted in encryptton unit 26 and stored on a CD storage 
medium 10 atong with the watermarked BASE image 1^ 
in unencrypted fomr) in additton to the original unencrypt- 
ed BASE/4 and BASE/16 images. Additionally, a 16 
BASE residual is formed and added to the CD storage 
medium 10. Note that except for a negligible increase in 
the size of the compressed 4BASE residual, there are 
no addittonal menDory requirements since the watermark 
removal component is embedded in the 4BASE residual. 

Throughout this descriptton of the invention refer- 
ence is made to the forming, merging, or differencing of 
images. It is understood by those skilled in the art that 
the pixel values representing images are manipulated, 
not the images themselves. 

For browsing or proofing, the PRIOR ART system of 
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Figure 2 is employed The PRIOR ART system arrange- 
ment of Figure 2 is substantially the same as the arrange- 
ment illustrated in Figure 3 of US. Patent Application Se- 
rial No. 146,371, referenced above. The user retrieves 
the BASE/1 6, BASE/4, or watemrarked B/VSE images di- s 
rectly without decryption from the CD storage medium 
10. Upon authorization, the user inputs a decryption key 
to the data decryption unit 28 to decrypt the modified 
4BASE residual. The watermarked BASE image is de- 
coded in decoder 30 and interpolated using linear inter- io 
polation and is added to the decrypted nrKxiified 4BASE 
residual in the reconstnjction apparatus 201 to recover 
the 4BASE image. Details of the reconstruction appara- 
tus 201 can be obtained from Figure 2, and the associ- 
ated description found in US. Patent No. 4,969,204, In 
general, rf the modified 4BASE residual Is quantized prior 
to its storage on the CD 10 (which is usually the case), 
the exact 4BASE image cannot be recovered. The dis- 
crepancy between the original 4BASE image and the 
4BASE image recovered according to the above proce- 20 
dure depends on the coarseness of the quantizer in ad- 
dition to the nature of the watermark. The quantizers 
used in the PRIOR ART systems of Figures 1 and 2 are 
part of the encoders 20 

The descriptk>n of the problem encountered with the 2S 
PRIOR ART systems of Figures 1 and 2 can be clarified 
by using an example. Assuming that an image is repre- 
sented by a two-dimensbnal array of pixels with the pix- 
els being represented by numbers with indk:es (i,j) de- 
noting the kx^ation of a pixel at the ith row and the jth ^ 
column , where i=l. ... , N,, and i=l, . . Np. and N^ and Np are, 
respectively, the number of the lines and the number of 
pixels per line in the image, ar>d that the brightness of 
each pixel is represented by an B-bit number, i.e., a 
number in the range of 0 to 255, Now conskier an exam- 35 
pie where, for the purpose of simplicity, a low-resolutk)n 
image, hereafter referred to as the BASE image, is com- 
posed of a 5x5 array of pixels as illustrated in Figure 3. 
Referring to Box 34 in Figure 1 , a watermark W is added 
to this image to aeale the watermarked B/VSE Image of 40 
Figure 4. The continuous line m Figure 4 shows the re- 
gion affected by the watermark. In this example, the ef- 
fect of the watermark is to replace the value of the pixels 
overlaid by the watermark W by a unifomi brightness val- 
ue of 200 so as to create a high contrast between the 45 
watermark area arwi the remaining pixels in the BASE 
image. 

Figure 5, hereafter referred to as the 4BASE image, 
shows an image that consists of a 9x9 array of pixels 
corresponding to the high resolution rendition of Figure so 
3. Referring to Box 24 in Figure 1, the watermariced 
BASE image in Figure 4 is interpwlated to create a pre- 
dictkxi for the 4BASE image. The interpolation scheme 
used in this example is linear interpolatbn and is de- 
scribed with reference to Figure 6. For best system per- ss 
formance the interpolator 24 should match the Interpo- 
lator 49 (shown in Figures 16, 17,and18). Thefourpbcels 
in the kw-resolutkxi image labeled through X4 are 



linearly interpolated to create the pixels through Y3 
according to the following relationships: 

Y,= (X,+X2)/2, 
Y2- (X,+X3)/2, 

^3= (X^+X2+X3+X4)/4, 

Where, in order to insure that the Yj values are also 8-bit 
integers, all fractions are rounded to the nearest integer 
and a fractional value of 0.5 is truncated down, e.g„ a 
value of 58.50 is truncated to the integervalueof 58. Fig- 
ure 7 illustrates the interpolated watermarked BASE im- 
age according to the above interpolation scheme where 
the boxes highlight the pixels that correspond to the 
low-resolution image (watermarked BASE image) of Fig- 
ure4. Referringto the subtracter 32, in Figure 1, the mod- 
ified 4BASE residual image is formed as the pixel-by-pix- 
el difference of the 4BASE image of Figure 5 and the 
interpolated watermarked BASE image of Figure 7. Fig- 
ure 8 illustrates this reskJual image which is the input to 
Box 20 in Figure 1 . It can be easily seen that if the resid- 
ual inr^age of Figure 8 is stored on the CD storage medi- 
um 10 without any quantizatkxi, the original 4BASE im- 
age can be completely recovered. However, in nxjst ap- 
plications, to save storage space, the residual image is 
quantized prtor to its storage, 

An example of the characteristics of a quantizer 
used in conjunction with the Kodak Photo CD system is 
shown in Figure 10. For this example, if the input to the 
quantizer is any number in the range of -47 to -255, it will 
be quantized (clipped) to the value of -48. Using the 
quantizer in Figure 10 to quantize the reskJual innage of 
Figure 8 results in the quantized residual shown in Figure 
9 whrch is then entropy encoded (e.g., Huffman encod- 
ed) and stored on the CD storage medium 1 0. /Vs shown 
in Figure 2, the 4BASE image is recovered by adding the 
quantized residual of Figure 9 to the interpolated water- 
marked BASE image of Figure 7. This results in the re- 
constructed 4BASE imago shown in Figure 11. Due to 
the errors introduced by the quantizer, the 4B/^E image 
reconstmcted in this way is sign ifk:antly different from the 
original 4BASE inrege especially in the regkxis affected 
by the watermark. The underlined pixels in Figure 11 are 
those that substantially differ from the orighal 4BASE im- 
age of Figure 5. The standard deviatkxi of the pix- 
el-by-pixel error between the original 4BASE image and 
the reconstructed 4BASE irrrage, also referred to as the 
root-mean-squared-error (RMSE), is computed to be 
63.89. A large portion of this error results from the fact 
that the quantizer clips the value of any negative reskJual 
with a magnitude larger than 48. If this reconstructed 
4BASE image is printed with a hard copy device or 
viewed on a monitor, it woukj display a visible trace of 
the watemriark. 

Referring to Figure 1 2, in accordance with the teach- 
ings of the present invention, the watenmarked tow-res- 
olution image lyy at the output of Box 34 is nrxxlified in an 
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image modifying urjit 36 prior to rts interpolation and stor- 
age on ttie CD storage nnedium in such a way as to re- 
duce the artifacts that result from the quantization of the 
modified residual in the encoder unit 20. The extent of 
the nrwdification depends on the characteristics erf the s 
quantizer and the watermark. For example, for a fine 
quantizer which contains many quantization levels, the 
changes in the visual quality of the watermarked BASE 
image, as a result of the present invention, will be negli- 
gible. On the other hand, if a coarse quantizer with a high io 
degree of clipping is used atong with a high-contrast wa- 
termark, the changes will be more perceptible and will 
generally appear as a reduction in the watennark con- 
trast. However, it shoukl be noted that in almost all ap- 
piicatk>ns of interest, the benefits of recovering a high is 
quality high-resolutbn image far outweigh any disadvan- 
tages of reducing the watermark contrast. 

The present invention introduces a method whereby 
the pixel values in a low-resolution image are modified 
prk>r to their encoding and stora^ in order to reduce the 20 
quantizatbn artifacts in the higher-resolution image. The 
objective is to introduce the smallest possible magnitude 
changes in the low-resolutkxi image pixel values that will 
also minimize the quantization error in the reskJual image 
while nr^aintainingthe viewability of the low-resolutkin im- 26 
age. In this way, the fidelity of the reconstnjcted high-res- 
olutbn image can be improved while minimizing any 
compromises to the image quality of the low-resolutbn 
image. In one embodiment of the present inventkxi, this 
techriique can be used to improve the reconstructbn 30 
quality of a high-resolution image in a hierarchk^ai image 
storage and retrieval system at the expense of introduc- 
ing minor changes in the tow-resolutkDn image. This is 
partkjularlyusefulforthoseapplicatbns where the image 
quality of a higher resolutkxi image component is more 35 
critical than the image quality of the lower resolutkx} im- 
age component. 

Referring to the flow diagram of Figure 1 3, whk;h il- 
lustrates the f unctkxis perfonmed by the image modifying 
unit 36. the difference (reskJual) between the value of 4o 
each pixel in the tow-resolutkxi image and the value of 
its corresponding pixel in the high-resoluton image is 
found. In the PhotoCD example, where a high-resolutkxi 
image has twfce as many pixels in each dimenskxi as 
the tow-resolution image, the high-resolution pixel loca- 4S 
tkxi corresponding to the tow-resolutkxi pixel tocatkxi 
(i, j) is (2i-1 ,2j-l) and the difference is found as the output 
of subtractkxi operation perfomied at circle 50. The re- 
sidual value (difference) from circle 50 is then passed 
through quantizer 52 to output a quantized reskiual val- so 
ue. The functkxial characteristk;s of a quantizer are 
shown in the chart of Figure 10. This value is then com- 
pared, in circle 60, against the residual value from circle 
50 to determine if an error exists. If it does the low-res- 
olutkxi pixel value is adjusted at f unclton 58, to make the ss 
quantizatkxi error zero. If no error exists no adjustment 
is required as stated in block 56. Each low resolutkx) pix- 
el value from 56 or 58 is stored in a storage buffer 70 



pending the adjustment of all pbcels forming the image 
If a second pass is to be implemented it uses the pixel 
values stored in storage buffer 70. If a second pass is 
not to be implemented the pixels from storage buffer 70 
may be output to the CD 10 and to the interpolator 24. 

Carrying further the example discussed prevtously, 
the reskiuats between the pixels in the watermarked 
BASE image of Figure 4 and their corresponding pixels 
in the 4BASE image of Figure 5 are found and adjusted 
according to the above scheme. The result after a first 
pass is shown in Figure 14. For example, the result of 
subtracting the pixel value in kx^tbn (1.1) of Figure 4 
from its corresponding pixel value in locatbn (1 ,1 ) of Fig- 
ure 5 is +1. Referring to the quantizer characteristics of 
Figure 1 0, a reskiual value of +1 is quantized to the value 
+1 resulting in no error and thus no adjustment for that 
pixel. As another example, the result of subtracting the 
pixel value in kx5atk>n (4.3) of Figure 4 from its corre- 
sponding pixel value in location (7,5) of Figure 5 is -132 
whrch is quantized to the value -48. The smallest mag- 
nitude adjustment to the tow-resolutran pixel value that 
will reduce the quantizatkxi error to zero is -84. Thus the 
pixel value in location (4.3) of Figure 4 is changed from 
200 to 116. As is seen from Figure 14 the effect of this 
process is to reduce the watermark contrast in the wa- 
temnarked BASE image. However, the quality of the re- 
constructed high-resolutk>n image is significantly im- 
proved. 

Figure 15 illustrates the reconstructed 4BASE im- 
age corresponding to the modified watermarked BASE 
image of Figure 14. Again, the underlined pixels differ 
from the original 4BASE image as a result of the quan- 
tizatkm error. The RMSE between the original 4BASE 
image and the reconstructed 4BASE image of Figure 15 
is 3.09. This constitutes a significant improvement com- 
pared to the RMSE of 63.89 corresponding to the recon- 
structed 4BASE image of Figure 11 . As is seen from Fig- 
ure 15, a large portion of this error is due to the three 
pixels In locatkxis (2,7), (2.8). and (2,9). whch have been 
highlighted 01 a box. In the present inventkxi, in order to 
further reduce the quantization error, the tow resolutton 
pixel values are adjusted In a second pass which con- 
sists of three steps as outlined bek>w. In describing the 
vartous steps of the second pass, reference is made to 
Figure 6 which illustrates a neighborhood containing the 
four k)w-resolutkxi pixels labeled through X4 and the 
resulting linearly interpolated pixels through Y3. 

Starting from the top-left pixel of the k>w-resolutk>n 
image resulting from the first pass stored in storage buff- 
er 70, and scanning the image from left to right and from 
top to bottom in a raster fashkxi, STEP(S) 1, 2 and 3 
described below are applied in successkxi to every pixel 
in the low-resolution image with the latest adjustment 
overwriting the previous pixel value in the storage buffer 
70; 

STEP 1 : For a given tow-resolutton pixel value X^, 
the interpolated pixel value is computed according to 
the interpolatton equation and its corresponding reskiual 
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value is found. The value of the low-resolution pixel X2 
(to the right of X^) is then adjusted by the smallest mag- 
nitude needed to eliminate the quantization error of that 
residual. Note that as a result of this adjustment, the re- 
sidual may sometimes be quantized to a value different 5 
from what it was quantized to prior to the adjustment. 
This is also true for STEP(S) 2 and 3 of the algorithm. A 
flow diagram describing the procedure employed in 
STEP 1 is depicted in Figure 16. 

STEP 2: For the same low-resolution pixel value ic 
as in STEP 1 , the interpolated pixel value is computed 
according to the interpolation equation and its corre- 
sponding residual value is found The value of the low 
resolution pixel X3 (below X-,) is then adjusted by the 
smallest magnitude needed to eliminate the quantization is 
error of that residual. A flow diagram describing the pro- 
cedure employed in STEP 2 is depicted in Figure 17. 

STEP 3: For the same low-resolution pixel value X^ 
as in STEP 1 , the interpolated pixel value Y3 is computed 
according to the interpolation equation and its corre- 20 
sponding residual value is found. The value of the low 
resolution pixel X4 (to the ri^t and below of X^) is then 
adjusted by the smallest magnitude needed to eliminate 
the quantization error of that residual. A flow diagram de- 
scribing the procedure employed in STEP 3 is depicted 25 
in Figure 16. 

It should be noted that at any step of the second 
pass, the calculations are based on the rTKxJified low-res- 
olution pixel values that reflect all the adjustments up to 
that stage. Also, each pixel in the low-resolution image 30 
may be adjusted up to three times during the second 
pass. For those pwels that are located on the image bor- 
ders, only a part of the neighborhood as defined in Figure 
6 might exist. In such cases, only those steps for which 
a neighboriiood is defined are earned out. The modified 35 
low-resolution image after the completion of the second 
pass would usually generate a very high-quality recon- 
stmction of the high-resolution image. 

The application of the second pass of the present 
invention to the numerical example cfiscussed before will 40 
help to illustrate further the operation of each step. Scan- 
ning the image of Rgure 1 4 in a raster fashion, it is found 
that no adjustment is necessary until the pixel location 
(1 .3) is reached. Denoting the value of this pixel by X^ , 
the value of the interpolated pixel in STEP 1 is found 45 
to be 92. The corresponding high-resolution pixel in Fig- 
ure 5 is located at (1,6) and has a value of 70. The re- 
sidual is thus -22 which results in no quantization error 
based on the quantizer characteristics of r=^igure 1 0. Sim- 
ilarly, the residual, corresponding to the interpolated val- so 
ue Y2 in STEP 2. is +1 and results in no quantization 
error. However, the value of Y3 in STEP 3 is found to be 
92, which, once subtracted from the value 76 located at 
position (2,6) in Figure 5, results in a residual of -17. This 
residual is quantized to a value of -16. introducing a ss 
quantization enor of -1 . The smallest magnitude adjust- 
ment needed to eliminate this quantization error will ad- 
just the pixel value X3, located at (2,4) in Figure 14, to 



119 instead of its current value of 118. After the adjust- 
ment, the residual value becomes -16 which is quantized 
without error. This procedure Is continued until all the low 
resolution pixel values are processed. In this example, 
the bw-resolution pixels that are located at (3,5). (4,2), 
(4,5), and (5,5), are adjusted three times during the sec- 
ond pass. 

Proceeding with the example, the completion of the 
second pass results in the modified watermarked BASE 
image of Figure 19. Note that while most pixel values in 
Figure 1 9 have undergone minor adjustments compared 
to their corresponding values In Figure 14. the two pixels 
in locations (2,4) and (2,5) have significantly changed. 
This was necessary to reduce the large quantization er- 
ror on the three pixels in the image of Figure 15 that are 
enclosed in a box. Figure 20 shows the reconstructed 
4BASE Image corresponding to the modified water- 
marked BASE image of Figure 19. Again, the underlined 
pixels differ from the original 4BASE image as a result 
of the quantization error. It can be seen that all the pixel 
values in Figure 20 are within one unit or less of their 
corresponding value in the original 4BASE image of Fig- 
ure 5 except for the pbcel located at (9,9), which shows 
a difference of 2 units. The RMSE between the original 
4BASE image and the reconstructed 4BASE image of 
Figure 20 is only 0.54, indicating the significant improve- 
ment resulting from the second pass. 

The high-resolution image, resulting from the 
present invention (Figure 20), compared with the PRIOR 
ART image of Figure 11, represents a significant im- 
provement. The price paid for this signrficant improve- 
ment is the replacement of the watenmarked BASE im- 
age erf Figure 4 with the iDodified watemnarked BASE im- 
age of Figure 19 whkDh has a reduced watermark con- 
trast. In most practical applications, the slight compro- 
mise in the usefulness of the low-resolutkxi image com- 
ponent is far outweighed by the image quality improve- 
ments in the high-resolution innage. 

The two-pass algorithm described atx>ve constitutes 
a preferred method for achieving the objectives of the 
present invention, namely, finding the sn^llest possible 
magnitude changes in the low-resolutkxi Image pixel val- 
ues that also aim at minimizing the quanttzatk>n error in 
the reskJual image. It shoukl be noted that several vari- 
atkxis tothis main theme are possible. In general, based 
on simulatkxis with the Photo CD image storage and re- 
trieval system, it was found that the preferred method 
resulted in a sli^tly better high-resolution image quality 
(for the same degree of nnodificatk>n in the tow-resolution 
image) compared to the other variations. However, in 
some applkiations. other system consideratbns might 
nr^ke the use of the alternative approaches more attrac- 
tive. In what follows, a few of these variations are briefly 
described with reference to the preferred method. 

In one variation, only the first pass of the algorithm, 
as described above, is implemented to create a reason- 
ably high-quality, high-resolutbn image and the second 
pass is omitted to reduce the computatkxial complexity. 
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In another variation, the first and the second passes are 
connbtned into a single pass in the following way. Starting 
from the top-left pixel of the low-resolution image, the val- 
ue of the pixel is first modified according to the teach- 
ings in the first pass of the preferred method. Next, the s 
values of the pixels Xg. X3, and are modified accord- 
ing to the STEP(S) 1 -3 of the second pass. Scanning the 
image from left to right and from top to bottom, this pro- 
cedure is repeated for every pixel In the low-resolution 
innage. Thevalueof each pixel can be modified up to four 10 
times according to this procedure. However, since the 
order in which the adjustment takes place is different 
from the preferred method, in general, a different image 
will result. In yet another variation, the first pass is imple- 
mented as described in the preferred method, but the is 
three steps in the second pass are implemented as three 
separate passes. That is, starting from the top-left pixel 
of the low-resolution image, the image is scanned from 
left to right and top to bottom and the STEP 1 of the sec- 
ond pass is applied to all the pixels. Then, the same pro- 20 
cedure is followed for STEP 2. Finally, STEP 3 is imple- 
mented in a similar fashion. Again, this results in a dif- 
ferent order in which the varbus pixels are processed. 
In yet another variation, after the execution of the first 
pass as described in the preferred method, the three 25 
steps in the second pass are carried out. but at each 
step, instead of adjusting the values of the pixels Xj, X3, 
or X4, the value of the pixel X^ is modified. Again, this 
will resuh in a different order of modifying the low-reso- 
lution pixel values than any of the schemes described so 
before. 

Furthermore, although the invention has been de- 
scribed with a watermark insertion unit which introduces 
high contrast edges, the man skilled in the art wouW re- 
alize that, as already mentioned eariier, for some imag- 35 
es, the use of the invention provides an improvement 
over the prbr art even when no watermark insertion unit 
is provkJed. 

Finally, while there has been shown what are con- 
sidered to be the preferred embodiments of the inven- 40 
tkxi, it will be manifest that many changes and modifica- 
tkxis may be made therein without departing from the 
essential spirit of the tfiventk)n. It is intended, therefore, 
in the annexed claims, to cover all such changes and 
modifbatkxis as may fall within the true scope of the in- 4S 
vention. 



Claims 

50 

1. A method for reducing quantizatbn artifacts in a 
hierarchbal image system of the type that decom- 
poses a high resolutkm image into low resolutbn 
image components, including low resolutkxi image 
pixel values, and resbual image pixel values com- ss 
prising the steps of: 

a) determining the difference values between 



the pixel values in the bw resolution image and 
the corresponding pixel values in the higher res- 
olution image; 

b) comparing each of the difference values 
against quantizer conversbn values to deter- 
mine if an error wou W exist for an executed con- 
version; 

c) adjusting the inr>age pixel values within the 
bw resolutbn image component if an error 
would exist, to a value that eliminates the error 
to form an adjusted tow resolution image; and 

d) using the unadjusted image pixel values as 
part of the adjusted tow resolutbn image com- 
ponent if an error does not exist. 

2. A method according to Claim 1 further comprising : 

e) forming a predicted higher resolution image 
by interpolating the pixel values of the adjusted 
bw resolution image components of steps c) 
and d); 

f) determining the difference values between 
the interpolated pbcel values in the predated 
higher resolution image and the corresponding 
pixel values in the higher resolutbn image; 

g) comparing each of the difference values 
against quantizer ccmversbn values to deter- 
mine if an error woukJ exist for an executed con- 
version; 

h) adjusting further the image pixel values within 
the adjusted tow resolution image component if 
an error woub exist, to a value that eliminates 
the error to fonm an adjusted low resolution 
image; and 

i) using the unadjusted image pixel value as part 
of the adjusted tow resolution image component 
if an error does not exist. 

3. A method for according to Claim 2. wherein a step 
of applying watermark pixel values to pixel positions 
within the tow resolutkxi image is perfonned before 
step a) and sakJ modified low resolution image is 
used as low resolutbn image. 

4. A method according to Claim 3, wherein the pre- 
dbted higher resolutbn inriage of step e) is formed 
by sequentially interpolating the pixel values of a first 
block d low resolution innage pixel values and steps 
f) through i) are repeated for each pixel in the higher 
resolution image. 

5. A storage device having stored thereon the adjusted 
low resolutbn image components and the residual 
image pixel values of any of Claims 1 to 4. 

6. Apparatus for reducing quantizatbn artifacts in a 
hierarchbal image system of the type that decom- 
poses a high resolutbn image into bw resolution 
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components, including low resolution image pixel 
values, and residual image pixel values, comprising; 

means for determining the difference values 
between the pixel values in the low resolution s 
image and the corresponding pixel values in the 
higher resolution image; 
means for comparing each of the difference val- 
ues against quantizer conversion values to 
determine if an en-or would exist for an executed 10 
conversion; and 

means for adjusting the image pixel values 
within the low resolution image component if an 
error would exist, to a value that eliminates the 
error to form an adjusted bw resolution image '5 
component and using the image pixel value as 
part of the adjusted low resolution image com- 
ponent if an error does not exist. 

Apparatus according to Claim 6 further comprising : 20 

means for forming a predicted higher resolution 
image by interpolating the pixel values of the 
adjusted low resolution image component; 
means for determining the difference values 2S 
between the interpolated pixel values in the pre- 
dicted higher resolution image and the corre- 
sponding pixel values in the higher resolution 
image; 

means for comparing each of the difference val- 30 
ues against quantizer conversion values to 
determine if an en^or would exist for an executed 
conversion; 

means for adjusting the image pixel values 
within the adjusted low resolution image com- 35 
ponent if an error would exist, to a value that 
eliminates the error to form a twice adjusted low 
resolution image component; and 
means for using the Image pixel value as part 
of the twice adjusted low resolution image com- 40 
ponent if an error does not exist. 

Apparatus according to Claim 7, further comprising: 

means for applying pixel values to pixel posi- 4S 
tlons within the low resolution image to form a 
tow resolution Innage component. 
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HIGH-RESOLUTION 
PIXEL VALUE AT 
LOCATION Y3 FROM 

DECOMPOSITION 
APPARATUS 101 



RESIDUAL VALUE 



QUANTIZER 
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QUANTIZED RESIDUAL VALUE 
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QUANTIZATION ERROR 



QUANTIZATION ERROR=0 
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YES 



ADJUST THE LOW-RESOLU- 
TION PIXEL VALUE X4 BY 

THE SMALLEST MAGNITUDE 
CHANGE THAT WILL MAKE 
THE QUANTIZATION ERROR 
ZERO 
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NO ADJUSTMENT TO 
THE LOW-RESOLUTION 
PIXEL VALUE X4 IS 
REQUIRED 
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ADJUSTED LOW-RESOLUTION PIXEL VALUE 
TO STORAGE 70 



FIG. 18 
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70 74 66 118 115 

60 62 68 97 62 

62 65 70 112 112 

114 107 117 119 114 

111 110 112 116 112 
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